Field
The present disclosure generally relates to techniques for detecting malware in an executable. More specifically, embodiments presented herein provide techniques for detecting malware in a packed executable containing junk instructions used to disguise the detection of the packed executable.
Description of Related Art
Computer systems are continually threatened by attacks from malicious computer code or malware. Malware generally refers to applications installed on a computer without a user's knowledge or without a user's consent, such as a virus, a worm, or a Trojan. Antivirus prevention and detection software installed on computers can attempt to prevent malicious code attacks and to detect the presence of malicious code. However, as malware evolves, the antivirus software too has to evolve to keep up with the latest malware.
Signature-based malware detection methods face a problem with an antivirus detection evasion tool called a “packer.” A packer is a software application, which takes an existing piece of malware and hides or “packs” it to make it no longer detectable to most signature-based systems. A packer can change the byte-level representations of a binary program without modifying its execution semantics. Because signatures used in anti-virus scanning engines are typically derived from the byte-level representations of malware samples, malware writers use packers to hide the malware. Even worse, malware writers can also apply different packers in different combinations to create a large number of variants of existing malware that can easily evade signature-based anti-virus scanners.
Traditionally, anti-virus companies have attempted to manage the packer problem by unpacking packed binaries and scanning the unpacked data. Typically, well-packed data has high entropy relative to normal code. Static scanners can usually detect an encrypted program based on the entropy of the code section. Newer packers insert junk instructions in their code stream to make the entropy look more normal. Thus, the junk instructions will not trigger traditional quick encryption detection methods.
Junk instructions make unpacking the packed executable more difficult. It is harder to discriminate junk instructions from others. Malware authors rely on the scanning tool trying to unpack the packed executable with the junk instructions before malware is detected.